Method for Rendering an Augmented Object

ABSTRACT

The present disclosure describes a new method for rendering ray traced reflections, applied to augmented reality and virtual reality. The intersections between secondary rays and scene geometry are done in large groups of rays, gaining high speed and lowering the computational complexity. Its reduced power consumption is suitable to consumer class of computing devices.

CROSS-REFERENCE TO RELATED CASES

The present application is a Continuation application of the U.S.application Ser. No. 15/615,037 filed Jun. 6, 2017 entitled “Method forRay Tracing Augmented Objects”; and it claims priority based on U.S.Provisional Application No. 62/354,755 filed Jun. 26, 2016 entitled“Projection Based Bouncing of Secondary Rays”; and claims priority basedon U.S. Provisional Application No. 62/408,730 filed Oct. 15, 2016entitled “Path Tracing by Separate Trajectory Systems”; and is aContinuation-In-Part of the U.S. application Ser. No. 15/376,580 filedDec. 12, 2016 entitled “Path Tracing Method Implemented On Cells AndEmploying Distributed Acceleration Structures”; and is aContinuation-In-Part of the U.S. application Ser. No. 15/009,442 filedJan. 28, 2016 entitled “Shadowing Method for Ray Tracing Based onGeometrical Stencils”; all of which are hereby incorporated byreference.

FIELD OF THE INVENTION

The present invention relates to new and improved ways of carrying outray tracing method at a reduced computational complexity.

BACKGROUND OF THE INVENTION

Augmented reality (AR) is a live view of a physical, real-worldenvironment whose elements are augmented by computer-generated graphics.Information about the augmented objects is overlaid onto the real world.Augmentation is conventionally in real time and in visual context withenvironmental elements. With the help of advanced AR technology, theinformation about the surrounding real world of the user becomesinteractive and digitally manipulable. Because AR brings out thecomponents of the digital world into a person's perceived real world,the user experiences and interacts with the augmented reality, dependingon the level of photo-realism of the augmented objects and theirintegration with the real environment.

In prior art, the graphics used in AR were based on the conventionalraster technology. However, raster graphics are very mediocre due to thetradeoff of quality and speed, in favor of speed. Its visual contextlacks such necessary elements of photo-realism as reflections,refractions, color bleeding, color bleeding, caustics, etc. In priorart, a high quality computer-generated realism is found in the filmindustry, enabled through the use of ray tracing computer-generatedgraphics.

Ray tracing is a computer graphics technique for generating images bytracing the rays of light and simulating the effects of their encounterswith virtual objects. The idea behind ray-tracing is to findmathematical solutions to compute the intersection of a ray with varioustypes of geometry, by solving the visibility between points. Thistechnique is capable of producing a high visual realism. In low-end raytracing, light sources are the only illuminating objects in the scene.

More realistic imaging is delivered by a high-end ray tracing, calledpath tracing, which is based on global illumination. Global illuminationtakes into account not only the light which comes directly from a lightsource, but also subsequent cases in which light is returned by othersurfaces in the scene, whether reflective or not.

Path tracing, referred to as Monte Carlo ray tracing, renders a 3D sceneby randomly tracing samples of possible light paths. Repeated samplingfor any given pixel in the image will eventually cause the average ofthe samples to converge to the correct solution of a rendering equation,making it one of the most physically accurate 3D graphic renderingmethods in existence. Path tracing can generate images that are faithfulto reality, and are indistinguishable from photographs (e.g. The Avatarmovie). The visual quality is higher than that of ray tracing, but at amuch greater computational cost.

The most time-consuming tasks in the ray tracing of prior art aretraversals of acceleration structures, as well as intersection testsbetween rays and polygons. Every single ray is traversed across anaccelerating structure (e.g. K-trees or BVH trees), seeking polygonsthat are candidates for intersection. These traversals become a majortime-consuming action—they typically take 60%-70% of the imagegeneration time. Then, all candidate polygons associated with the searchmust undergo a line-triangle intersection test, to determine theearliest hit along the ray's path.

The layout of the prior art ray tracing method is depicted in FIG. 1.First, an acceleration structure must be constructed 10. Theconstruction is done as a preprocessing step, and takes much more timethan generating a single image. Generally, the construction time dependson the scene size. The bigger the scene, the longer the constructiontime. Every major modification in the scene necessitates areconstruction of the acceleration structure. The memory size istypically doubled by the acceleration structure. Tracing of rays 12 isbased on massive traversals of the acceleration structure 11, when eachray is traversed across the structure in search of intersections betweenthe ray and various scene objects. The resulting intersection points arelighted, textured, shaded 13, and aggregated into image pixels.

There are two major drawbacks associated with the use of accelerationstructures in ray tracing of prior art; (i) they must be repeatedlyreconstructed for scene changes, and (ii) traversals of these structuresare time-consuming. Both disadvantages contradict with the real-timerequirements of AR.

Therefore, the primary object of the present invention is to acceleratethe performance of global illumination ray tracing up to real time,making it suitable for AR.

Another object of the present invention is to reduce the computationalcomplexity of ray tracing.

Another object of the present invention is to reduce the powerconsumption of ray tracing.

Another object of the present invention is to enable global illuminationray tracing by the processing level of consumer computing devices.

SUMMARY OF THE PRESENT INVENTION

Some embodiments of the current invention are applied to both augmentedreality (AR) and virtual reality (VR). AR is a live view of a physical,real-world environment whose elements are augmented bycomputer-generated graphics. VR replaces the real world with a simulatedone. Augmentation is conventionally in real time and in a visual contextwith environmental elements. The user's experience and interaction aredirectly influenced by the level of realism in the AR and VR.

In the prior art, the imaging of augmented objects is produced by theconventional raster graphics, due to its high speed. With the help ofadvanced AR technology (e.g. adding computer vision and objectrecognition), the information about the surrounding real world of theuser becomes interactive and digitally manipulable. Information aboutthe environment and its objects is overlaid onto the real world.

However, the ordinary raster graphics technology is mediocre in imagequality and in visual context with the real-world environment. Thedesired high quality of computer-generated realism can be found today inthe film industry, and is enabled through the use of the globalillumination ray tracing, namely path tracing. Unfortunately, it doesnot fit the AR because of a very high computational complexity, whichcauses long production times and requires expensive computing farms.

The present invention teaches an innovative way of delivering pathtracing in real time, having reduced computational complexity and powerconsumption, and suitable to the processing level of consumer computingdevices. Aspects of the present invention enable to concentrate onselected object(s), generating a photo-realistic image of the objectrealistically overlaid with a preset environment.

In the embodiments of the present invention, the Acceleration Structuresof prior art are replaced by a new and novel device—a DynamicallyAligned Structure (DAS). DAS is a means for carrying out theintersection between the secondary rays and the scene geometry in largegroups of rays, gaining a high speed and a reduced computationalcomplexity.

FIG. 2 shows the stages of the present invention's path tracing. Themain difference as compared to the prior art (FIG. 1) is the lack ofacceleration structures. These structures are replaced by the DAS device21. There is no pre-processing for reconstruction, and no traversals ofacceleration structures.

The DAS is an aligned projection of rays, used to carry secondary raysassociated with existing hit points. Instead of an individual shootingof secondary rays for every single hit point (or in small packets ofrays), as in the prior art, we do it collectively, cutting the cost.

BRIEF DESCRIPTION OF THE DRAWINGS

The invention is herein described, by way of non-limiting examples only,with reference to the accompanying figures and drawings, wherein likedesignations denote like elements. Understanding that these drawingsonly provide information concerning typical embodiments of the inventionand are not therefore to be considered limiting in scope:

FIG. 1. Prior art. Block diagram of ray tracing.

FIG. 2. Block diagram of ray tracing of the present invention.

FIG. 3. Prior art. Secondary rays generating global illumination at apoint of ray/object intersection.

FIG. 4. The basic mechanism of DAS.

FIG. 5. Multiple DAS projections shot in randomly varying directions.

FIG. 6. Generating hit points by successive DAS projections.

FIG. 7a . Opening scene consisting of two triangles and two primaryHIPs.

FIG. 7b . An early segment of a DAS ray shot towards a HIP.

FIG. 7c . A main segment of a DAS ray carries a secondary ray, shot fromthe HIP.

FIG. 8a . DAS renders HIPs data only.

FIG. 8b . DAS renders the scene geometry. Objects along the earlysegment of DAS ray are discarded.

FIG. 8c . DAS flowchart.

FIG. 9. Various cases of secondary rays carried by a single DASprojection

FIG. 10. An augmented object standing on a real desk of asemi-reflective surface.

FIG. 11. Direct imaging of augmented object. Primary rays are shot fromthe camera and tilted at varying directions.

FIG. 12a . Direct imaging of augmented object. Secondary rays aregenerated by multiple DAS shots of a perspective projection.

FIG. 12b . Direct imaging of augmented object. Secondary rays aregenerated by multiple DAS shots of parallel projection, saving renderingof a bigger data.

FIG. 12c . Flowchart of generating direct image of an augmented object.

FIG. 13a . Reflected imaging of augmented object. Primary rays are shotat the reflection area, repeated and tilted multiple times.

FIG. 13b . Reflected imaging of the augmented object. The secondaryrays, carried by DAS projection, are shot at the object through thecluster of primary HIPs.

FIG. 14a . Reflected imaging of the augmented object. The multiple DASprojections are randomly tilted.

FIG. 14b . Reflected imaging of the augmented object. The contributionof a secondary ray to the aggregated light energy comply with the BRDFfunction.

FIG. 14c . Flowchart of reflected imaging.

FIG. 15a . Color bleeding effect of the augmented object on itsenvironment. Primary rays.

FIG. 15b . Color bleeding effect of the augmented object on itsenvironment. Secondary rays.

FIG. 15c . Flowchart of generating color bleeding effect.

FIG. 16. Collecting the sampled light values at the pixel of origin.

FIG. 17. Hardware for AR and VR.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

The principles and operation of an apparatus according to the presentinvention may be understood with reference to the figures and theaccompanying description wherein similar components appearing indifferent figures are denoted by identical reference numerals. Thedrawings and descriptions are conceptual only. In actual practice, asingle component can implement one or more functions; alternatively,each function can be implemented by a plurality of components anddevices. It will be readily understood that the components of thepresent invention, as generally described and illustrated in the figuresherein, could be arranged and designed in a wide variety of differentconfigurations. Thus, the following more detailed description of theembodiments of the apparatus, system, and method of the presentinvention, as represented in the figures herein, is not intended tolimit the scope of the invention, as claimed, but is merelyrepresentative of embodiments of the invention.

Unless specifically stated otherwise, as apparent from the followingdiscussions, it is appreciated that throughout the specificationdiscussions, utilizing terms such as “processing”, “computing”,“calculating”, “determining”, “generating”, “creating” or the like,refer to the action and/or processes of a computer or computing system,or processor or similar electronic computing device, that manipulateand/or transform data represented as physical, such as electronic,quantities within the computing system's registers and/or memories intoother data, similarly represented as physical quantities within thecomputing system's memories, registers or other such informationstorage, transmission or display devices.

Embodiments of the present invention may use terms such as processor,computer, apparatus, system, sub-system, module, unit, and device (insingle or plural form) for performing the operations herein. This may bespecially constructed for the desired purposes, or it may contain ageneral purpose computer selectively activated or reconfigured by acomputer program stored in the computer. Several technical termsspecifically associated with our disclosure are herein defined.

Computer graphics pipeline refers to the most common form of computer 3Drendering, 3D polygon rendering, distinct from raytracing, andraycasting. In particular, in raycasting, a ray originates at the pointwhere the camera resides, if that ray hits a surface, then the color andlighting of the point on the surface where the ray hit is calculated. In3D polygon rendering the reverse happens, the area that is in view ofthe camera is calculated, and then rays are created from every part ofevery surface in view of the camera and traced back to the camera. Thegraphics pipeline is usually used in real-time rendering.

Rendering a projection is a 3D computer graphics process ofautomatically converting 3D wire frame models into 2D images renderingon a computer. The projection can be of a perspective, parallel, inverseor of another shape.

Render target is a feature of modern graphics processing units (GPUs)that allows a 3D scene to be rendered to an intermediate memory buffer,or Render Target Texture (RTT), instead of the frame buffer or backbuffer. This RTT can then be manipulated by pixel shaders in order tomake searches or apply effects to the final image.

Primary rays, are the first generation of rays in ray tracing, cast intothe scene from camera or from eye to solve for visibility, i.e. to findwhether the primary ray intersects a surface.

Secondary rays in ray tracing are spawned from primary rays at theyray-polygon intersection points. They are used to compute things likeshadows, reflections, refractions, etc. Herein, we use the termcollectively for all successive generations as well, such as for ternaryrays, forth generation, etc.

Global illumination, is a general name of a group of algorithms used in3D computer graphics that are meant to add more realistic lighting to 3Dscenes, taking into account not only the light that comes directly froma light source (direct illumination), but also subsequent cases in whichlight rays from the same source are reflected by other surfaces in thescene, whether reflective or not (indirect Ilumination).

Color bleeding in computer graphics is the phenomenon in which objectsor surfaces are colored by reflection of indirect light from nearbysurfaces. This is a visible effect that appears when a scene is renderedwith full global illumination.

Accelerating structures, such as grids, octrees, binary spacepartitioning trees (BSP trees), kd-trees and BVHs (bounding volumehierarchy), are used in ray tracing to solve for visibility, allowingimprovement of render times in speed and efficiency, as compared tonaïve ray tracing without accelerating structures.

GPGPU (general-purpose computing on graphics processing units) is theuse of a graphics processing unit (GPU), which typically handlescomputation only for computer graphics, to perform computation inapplications traditionally handled by the central processing unit (CPU).

Preset scene in AR, replaces the real time world. It is a preprocessedenvironment scene to contain the augmented object.

Object can stand for a primitive (polygon, triangle, solid, etc.), or acomplex object made out of primitives.

Hit point—is a point where a ray intersects an object. Termed also HIP.

Visibility—given a set of obstacles in the Euclidean space, two pointsin the space are said to be visible to each other, if the line segmentthat joins them does not intersect any obstacles.

Scene, a collection of 3D models and lightsources in world space, intowhich a camera may be placed, describing a scene for 3D rendering. Scenemodel elements include geometric primitives: points or vertices; linesegments or edges; polygons or faces.

Clipping, in the context of computer graphics, is a method toselectively enable or disable rendering operations within a definedregion of interest.

The processes/devices and displays presented herein are not inherentlyrelated to any particular computer or other apparatus, unlessspecifically stated otherwise. Various general purpose systems may beused with programs in accordance with the teachings herein, or it mayprove convenient to construct a more specialized apparatus to performthe desired method. The desired structure for a variety of these systemswill appear in the description below. In addition, embodiments of thepresent invention are not described with reference to any particularprogramming language. It will be appreciated that a variety ofprogramming languages may be used to implement the teachings of theinventions as described herein.

Diverting from prior art technologies, aspects of the present inventionteach how to implement ray tracing at both a reduced computationalcomplexity and a high speed. One aspect of the invention relates to pathtracing, which is a high-quality ray tracing based on globalillumination. Its superior performance stems from a differenttechnological approach to solving the intersection between rays andscene objects. It is based on dynamically aligned structure (DAS), whichis a projection of parallel rays, used to carry secondary rays emittingfrom existing hit points. The DAS mechanism can be implemented either bya GPU (graphics processing unit) graphics pipeline, or by a CPU (centralprocessing unit). The mechanism can solve ray-triangle intersections byuse of the conventional graphics mechanism, replacing the expensivetraversals of accelerating structures in the prior art.

DAS mechanism. In one embodiment the DAS mechanism is applied to pathtracing, which is based on global illumination. Global illumination (orindirect illumination) takes into account not only the light that comesdirectly from a light source, but also light reflected by surfaces inthe scene, whether specular, diffuse, or semi-reflective. FIG. 3 depictsthe sampling of diffuse inter-reflection from the surroundingenvironment, at a given surface point. In order to achieve globalillumination on a diffuse surface, sampling rays must be shot from a hitpoint (HIP) 33. HIP is a result of a previous encounter between a ray(primary or secondary) and a triangle. The sampling is done by shootinga ray in a random direction within the boundaries of a hemisphere 31.The hemisphere is oriented such that its north pole is aligned with thesurface normal.

The basic mechanism of DAS is demonstrated in FIG. 4, where it is shownto be associated with a perspective projection; however, otherprojections such as parallel or inverse are conceivable as well. The DASstructure comprises a projection of rays passing through a cluster ofHIPs, e.g. 403, 405, 408, and targeting an object. The DAS rays that runinto HIPs are used as carriers for secondary rays. For example, a DASray that incidentally runs into a HIP 408 carries a secondary ray 406.This ray is falling within the HIP's hemisphere 407. A DAS ray that runsinto a HIP or a triangle consists of a starting point 400, and may havetwo intersection points, the first with a HIP 408 and the second with ascene object (triangle) 409. Additional secondary rays associated withthe same HIP 408 can be independently generated by additional DASstructures, carrying additional secondary rays in other directions,within the same hemisphere.

According to an embodiment of the present invention, the DAS projectioncan be implemented by a CPU software graphics pipeline, but the mostefficient device is the GPU hardware graphics pipeline. This well-knownart computer graphics pipeline is the process of turning a 3D model intowhat the computer displays. A graphics pipeline consists of twosubsystems: geometry and rasterization. First, all objects within theDAS frustum are transformed by the geometry sub-system, according to thecamera view. Then, in the raster sub-system, a ray/triangle intersectionpoint is selected by the Z-buffering mechanism. For example, the DAS ray402 in FIG. 4 is shot from the origin of the projection 400 andintersects with two objects; 408 and 409. Which of these two objects isselected, depends on the API directives (Direct 3D or OpenGL) thatcontrol the Z-buffering.

An exemplary cluster of three existing HIPs, with their underlyingtriangles, are shown, 405, 408 and 403. Secondary rays of HIPs 405, 408are driven by the DAS structure. As an example, the carrier ray 402 runsinto HIP 408. From the point of encounter with the HIP and onwards, itbecomes a secondary ray 406, associated with the HIP, seeking forintersection 409. A DAS is relevant only to HIPs that have theirhemispheres oriented toward the projection, like 405 and 408, but not403. The DAS method is mathematically articulated as follows:

Let T be a tree-graph of d levels and let V be its vertices on top ofgeometries G in space.

Define V_(d)—vertices within V in level d.

Let C_(d) be a division of V_(d) to clusters.

We shall extend T to d+1 levels by finding V_(d+1):

Choose cluster c ϵ C_(d), with V_(d) _(c) vertices and define L_(c)—setof mappings from V_(d) _(c) to V_(d+1) _(c) such that V_(d+1) _(c) areprojections of the vertices in V_(d) _(c) on top of G.

$V_{d + 1}:={\bigcup\limits_{c}V_{d + 1_{c}}}$

-   -   Note that L_(c) is a set of mappings from the same input, so        there can be several target vertices for any input vertex.

Instead of projecting every vertex v ϵ V_(d) _(c) on every possiblegeometry g ϵ G in θ(|L_(c) |·|V|·|G|), we project every possible g ϵ Gon every cluster c ϵ C_(d) in θ(|L_(c) |·(|V|+|G|)).

-   -   In R³ We can utilize traditional 3D graphics pipeline (raster        hardware) to achieve fast mappings (projections) in parallel.

We optimize C_(d)/L_(c) in throughput/overfitting to have:

-   -   Maximum number of vertices per cluster in average (throughput).    -   Minimum number of [discrete] projections of geometries fitting        all vertices (overfitting).    -   Preprocess/Runtime constraints.

L_(c) is chosen to have a pseudo-random output, representing a possiblesegment of distribution for each v ϵ V_(d) _(c) to simulate a physicalscenario.

Multiple DAS projections are shot at a scene or part thereof, inslightly different directions; each direction can be taken in a randommanner. As a result, multiple samples of the neighborhood can be takenat each HIP, for global illumination. This is illustrated in FIG. 5. HIP507 is a primary hit point created by the primary ray 508, shot from theimage pixel 500. The HIP 507 is visited by three subsequent DASprojections—501, 502 and 503. Each DAS carries one secondary ray for theHIP 507. Each of the three secondary rays delivers a different samplefrom the environment, i.e. from surfaces 504, 505, and 506,respectively.

There may be various ways to make use of the HIPs that are generated bysuccessive DAS projections. According to one embodiment, shown in FIG.6, all the newly generated HIPs contribute their data. In this example,four successive DAS projections are used. Assuming that 601 is a primaryHIP, generated previously by a primary shot from the image pixel 600,its first successor HIP 602 is a product of the first DAS projection.The second DAS projection, having a different direction, generates HIPchildren 603 and 604. The third DAS projection generates the HIPs 605,606, and 607. And then, one child HIP 608 is generated as a result ofthe fourth DAS projection. The light sampling from all HIPs must beaveraged and converged to the correct solution of rendering equation forthe image pixel 600, e.g. 608 and 605 converge into 603, which convergesinto 602, which in turn converges into 601. Finally, the primary HIP 601converges the aggregated sample values of all its children 602, 604, and607, and the result goes into pixel 600, as a partial contribution tothe pixel among other primary HIPs. A correct rendering equation for allconvergances should eventually generate a physically accurate image.

Secondary rays are meant to intersect with scene objects, asdemonstrated in FIGS. 7a-7c . FIG. 7a shows an initial scene consistingof two triangles 711, 712, and two primary HIPs 713, 714. In FIG. 7b theDAS projection 721 is shot towards the HIPs. The HIP 714 faces anopposite direction, therefore it is excluded from the current DAS. TheHIP 713, aligned positively with the projection, initiates a secondaryray. As shown further in FIG. 7c , the carrier ray associated with HIP713 is broken down into two parts; the early segment 731 and the mainsegment 732. The early segment 731 extends from the DAS origin up to theHIP 713. Its function is to pinpoint the HIP and its depth. Once thedepth Z_(HIP) is found, the main segment, from the HIP and forward tothe intersection point 733, plays the role of carrier for the secondaryray, searching for an intersection. It hits an object at 733, generatinga secondary HIP.

According to one embodiment of the present invention, the DAS projectionutilizes the Z-buffering mechanism of GPU, as illustrated in FIGS. 8aand 8b . The Z-buffering mechanism must discard objects all the waybefore the HIP, and starts seeking objects only from the HIP and on. Itis based on a selective use of the z-buffering mechanism of a GPU, e.g.the function glDepthMask of the graphics library of OpenGL. This is donein two separate rendering passes. In the first pass the HIPs arerendered as the only objects in the scene, disregarding the geometricdata and generating a HIP depth mask. In the second pass the HIP depthmask is used for rendering the scene geometry. The first pass is shownin FIG. 8a . The carrier ray 812, which overlays with the HIP 811, isbroken down into two segments, and is processed in two passes. The earlysegment, rendered during the first pass, extends from the camera 813 upto the HIP. The depth value of the HIP, Z_(HIP), is registered in theHIP depth mask 810. The depth value is kept for a later use forfiltering out all the objects on the way to the HIP, during the secondpass. In the second pass (FIG. 8b ), the geometric data is renderedstarting at the depth Z_(HIP), 811, e.g. the triangle 825 is ignored.The main segment, the carrier of the secondary ray, hits the triangle at823. The results of the second pass are stored in a render target 820.Rays that miss HIPs are entirely discarded, considered as early segmentsin their entirety. Once a render target is completed, the exactray/triangle intersection point 823 is found by inspecting the rendertarget at the u, v coordinates of the DAS carrier ray. The triangle ofintersection delivers essential data, such as color, light, normal,material, etc.

The DAS flowchart in FIG. 8c summarizes the method of generating andusing the DAS mechanism. The DAS projection targets an object (e.g. anaugmented object) or a sub-scene, passing through a cluster of HIPs, inorder to generate secondary rays for the HIPs. The DAS is shot twice.The first time it is shot at the HIP data only, ignoring the scenegeometric data, and producing a HIP depth mask 831. The second time anidentical DAS projection 832 is shot. This time the geometric data ofthe scene is rendered, ignoring the HIP data. The depth mask 810 isemployed to the starting point of secondary rays. Secondary rays aredriving on the DAS projection, seeking for intersection with thegeometric data. The rendering result, a render target, a 2D projectionof a 3D sub-scene, is basically a collection of all the intersectionpoints between the secondary rays and the geometric data of the scene.An intersection point, related directly to a specific HIP, can be foundby searching for coordinates u, v on the render target, that match thecoordinates u,v of the HIP. The color and light values at theintersection points are fed back to the HIPs, delivering samples ofglobal illumination 833. Finally, the intersection points are stored inthe HIP repository as a next generation of HIPs 834.

Various cases of secondary rays are shown In FIG. 9, all carried by asingle DAS projection. Ray 900 consists of two segments. The earliersegment, extending from the camera 909 up to HIP 903, discards thetriangle 906, while the main segment encounters the triangle 907 at theintersection point 905. The secondary segment of ray 902 does not hitany object. Ray 901 fails to encounter a primary HIP, therefore it isconsidered as an early segment in its entirety, ignoring the triangle908.

The DAS mechanism of the present invention is implementable, among otherfields, in AR. One of its embodiments enables a localized path tracing,focused on rendering of one or more objects in the scene, and on aperfect integration between the augmented object(s) and the realenvironment. FIG. 10 shows an example of an augmented object, a Buddhastatue 101, standing on a real desk of a semi-reflective surface 107.What is needed to generate a photo-realistic appearance of the statue isnot just the image of the object 101, but its reflection 102 as well.The effect the augmented object may have on its real environment wouldresult in reflections, shadows and color bleeding that modify the presetenvironment. On the other hand, the impact of the environment on theaugmented object may result in lighting and reflection of the objectitself.

According to an embodiment of the present invention, the image of theobject and the object's reflection within the environment are generatedby two separate tasks, and the unified result is fed to the imagepixels.

Direct imaging of an augmented object. The basic image of the augmentedobject can be reconstructed just from the primary HIPs that cover theobject's surface. However, for a global illumination effect on theimage, such as the environment reflected in the object, secondary raysare required, shot from the object to its environment. The renderingtask of the augmented object 110 is shown in FIG. 11. For simplicity, itis described in a 2D drawing. The camera 113 shoots primary rays 114 atthe augmented object, seeking intersections with the object. Thesepoints of intersection become HIPs, meant to be used as a starting pointfor secondary rays, for global illumination.

The shootings of primary rays repeat, each with a slight change ofdirection, such that multi-sampling in image pixels is attained. Thechange of directions is done in a random fashion, in order to preventunwanted patterns in the image. Multi-sampling contributes to anantialised quality of the image. In FIG. 11 three primary shootings areshown 115, 116, and 117.

A truthful and integrated appearance of the augmented object in thescene is achieved by global illumination. The relevant environment forglobal illumination is sampled by secondary rays, shot from primary HIPstoward a relevant part of the scene. The relevant parts of the scene arethose visible from the camera by reflection in the object, if the objectis reflective. For example, such a relevant part can be the sub-scene123 in FIG. 12a , because its reflection in the object can be seen fromthe camera 127.

The secondary rays are generated by DAS structures, either ofperspective projection as in FIG. 12a , or parallel projection as inFIG. 12b . In FIG. 12a the DAS projections are passing through theprimary HIPs (e.g. 128), targeting the sub-scene 123.

Since all the successive DAS projections are targeting the samesub-scene 123, this sub-scene can be clipped out from the full scene, toselectively enable rendering operations within a reduced region, thusminimizing the rendering process.

Each of the multiple DAS projections is generated randomly from aslightly different point of view and in a different direction, creatingmultiple secondary rays at each HIP. The use of randomness prevents theappearance of unwanted patterns in the image. Secondary rays sampleglobal illumination for a HIP (FIG. 3, 31), integrating between theobject and the environment. The sampled illumination impacts the imageaccording to the object's material and the level of its specularity ordiffuseness, e.g. if the object is reflective or partly reflective, itwill result in reflection of the environment in the object, or just anamount of background lighting if it is diffuse, creating the object'sresponse to the environment.

The more projections of DAS, the better covering of global illumination.However, more projections may impair the performance. Therefore, thereis a trade-off between image quality and performance.

The method of generating a direct image of an augmented object issummarized in a flowchart in FIG. 12c . First, multiple primaryprojections are shot from the camera (eye, view point) at the augmentedobject to generate a cluster of primary HIPs 1231. Then the parts of thescene to be targeted by secondary rays should be defined, possiblyclipped as a sub-scene 1232, and a reference point for DAS projectionsmust be set according to the chosen sub-scene 1236. Then, secondaryrays, generated by multiple DAS projections, are shot into the relevantsub-scene 1233. The result of the DAS projection is a render targettexture of the determined sub-scene. The seeking for an intersectionbetween a secondary ray and the determined sub-scene is done by matchingcoordinates of the related primary hit point and the render targettexture 1237. Each primary HIP is fed a light value of a correspondingintersection point between its secondary ray and the encounteredtriangle 1234. The above procedure can repeat if more than one sub-sceneare taken. Then, finally, the intersection points are added to the HIPrepository as a new generation of HIPs 1235. The processed samples ofcolor and light values, aggregated from all the primary hit points, areconverged into the image pixels, creating a full image of the augmentedobject affected by the three-dimensional scene.

Reflecting the augmented object. Reflection of the object's image in theenvironment items is accomplished by following a ray from the camera tothe surface at the scene, and then bouncing toward the augmented object.Reflection on a shiny surface or tile enhances the photo-realisticeffects of a 3D rendering. The extent of reflection depends on surface'sreflectivity (the BRDF of the material).

First, the reflective or semi-reflective surfaces (or items) in the realscene, which may reflect the augmented object, must be identified. Thenwe shoot primary rays at the surface in which the object is intended tobe reflected, or part thereof, generating primary HIPs. From these HIPs,we shoot secondary rays targeting and sampling the augmented object.This way of generating reflections is illustrated in FIGS. 13a and 13b .Primary HIPs covering the area of intended reflection are created byprimary rays that are shot from the camera 133, through the image screen130, toward the area of reflection 134. The location and boundaries ofthe reflection area 134 in surface 132 is determined according to thelocation of the camera, the distance and the size of the augmentedobject 110, and the consideration of the principal direction 131according to the Snell law. The primary shots are repeated multipletimes for multi-sampling in the image pixels. Each successive time theprimary projection is slightly deviated randomly from the principaldirection, such that each pixel of the image gets multiple samples. Thesurface of the reflection area 134 becomes covered by a dense array ofprimary HIPs. The randomness of the multi-sampling prevents unwantedpatterns in the resulting image.

FIG. 13b describes the generation of the reflected image by secondaryrays. The reflection of the augmented object 110 on the surface 132 isreconstructed from the sampled data in primary HIPs, which is gatheredby shooting secondary rays at the object. We use the geometric point136, which is the reflection of the camera 133 in the surface 132, as areference point for multiple DAS projections. Each projection is shotfrom a different point, randomly deviated from the reference point 136.

The DAS 135, shown in FIG. 13b , originates in the reference point 136,directed along the axis 139, which is pointing towards the center of theaugmented object 110. The DAS carries secondary rays 138, all startingat the primary HIPS (e.g. 137) and targeting the augmented object.

In order to take a spectral sampling of the BRDF function at a HIP, themultiple DAS projections are randomly deviated from the reference DASprojection—the one that starts at the reference point and has an axis ofprojection directed at the center of the augmented object. Theinclination from the referenced DAS is done randomly, slightly deviatingfrom the reference point 142 and central axis 145, as shown in FIG. 14a. Three DAS projections are shown. Assuming that the reference DASoriginates exactly at the reference point 142, and its axis 145 takesthe central direction, then the two other DAS projections start at anearby points 141 and 143, and their axes 144 and 146 deviate from thecentral direction 145. As an example, we chose a HIP 140 from whichthree secondary rays are shot: 144, 145, and 146, each carried by adifferent DAS.

The relation between the deviation of a DAS secondary ray from thereference DAS, and its contribution to the aggregated light energy, isshown in FIG. 14b . It is strongly connected with the BRDF function 147of the surface material 132. Each of the three secondary rays 144, 145,and 146, are shot from the same HIP in a different direction, bounded bythe hemisphere of FIG. 3. As a result, its sampled data contributes tothe aggregated light energy, according to the BRDF function. Let'sassume that the secondary ray 145 goes in an exact Snell direction, thenit brings the maximal contribution at the peak of the BRDF function 147.Secondary rays 144 and 145 have a smaller contribution, depending on theBRDF value at the distance from the peak.

The way of generating a reflected image of an augmented object issummarized in the flowchart in FIG. 14c . First, the area in the realscene where the augmented object should reflect is determined 1431.Then, multiple primary projections are shot from the camera at the areaof reflection, generating a cluster of primary HIPs 1432. Next, thelocation of the reflected camera, as a point of reference for DASprojections, and the central axis directed toward the augmented object,must be calculated 1433. Then, secondary rays generated by DAS are shottoward the object. The multiple DAS projections are randomly tilted,deviating from the reference DAS 1434. Next, the light values sampled inthe intersection points are fed to their respective HIPs of origin 1435.Finally, the intersection points are added to the HIP repository as anew generation of HIPs 1436. These HIPs can serve for producing furthergenerations of secondary rays.

Color bleeding is the phenomenon in which objects or surfaces arecolored by reflection of indirect light from nearby surfaces. It is aglobal illumination algorithm in the sense that the illuminationarriving at a surface comes not just directly from the light sources,but also from other surfaces reflecting light. Color bleeding isviewpoint-independent, which makes it useful for all viewpoints. Thecolor bleeding effect in AR or VR would occur in the direct neighborhoodof the augmented object. An example of generating a color bleedingeffect by an embodiment of the present invention is illustrated in FIG.15a . The augmented object 154, standing on a substrate 152, is supposedto create a color bleeding effect on the real substrate 152. First, wedefine the boundary of a color bleeding patch around the center of theaugmented object where the color bleeding will appear. The size of thepatch depends on the involved materials, and the distance and the amountof light. Then we shoot primary rays from the camera 153 at the patch155, in the absence of the augmented object. A cluster of primary HIPsis created, covering the patch. The primary shots are repeated multipletimes, each time at a slight deviation from the principal direction 151.The principal direction 156 is from the camera toward the center of theobject's standing location.

FIG. 15b illustrates the use of secondary rays. The color bleedingeffect is reconstructed by sampling the object by secondary rays, whichare shot from the primary HIPs toward the object. The secondary rays aregenerated by DAS projections. The DAS projection, unlike in the cases ofreflection or of direct imaging of the augmented object, gets the shapeof an inverse projection 156. Multiple DAS projections are done, eachtime at a slight deviation from the principal direction. The samples ofthe object's surface are taken from the substrate, enabling acalculation of the amount of energy on the substrate, assuming that acorrect rendering equation is used. The way of generating color bleedingis summarized in the flowchart of FIG. 15c . First, the location andsize of the color bleeding patch in the scene is defined 1531. Then,multiple primary projections are shot from the camera at the colorbleeding patch, to generate a cluster of primary HIPs 1532. Next, areference point at the center of DAS projection is calculated 1533, aswell as the required shape of the inverse projection 1534. Thensecondary rays are shot by multiple DAS projections, each randomlydeviated from the center of DAS projections 1535, and the light valuesof intersection points are fed to primary HIPs 1536. In color bleedingthis is the only generation of HIPs.

Collecting light values. The values of all samples in an HIP must beprocessed by a correct rendering equation for a physically accurateresult. Parameters that are taken into account are the surface material,the geometry of the scene, the active area of the hemisphere, andothers. For a specific image pixel, the light contribution of all HIPsgenerated by primary shots from the pixel, and all their secondarysuccessors, must be aggregated, processed, and converged into the sourcepixel for image. As shown in FIG. 16, the samplings from the object andfrom its environment are converged into image pixels 164. The pixelreceives inputs from a primary HIP 165 on the surface of the augmentedobject, which in turn collects values from its successive generations.The pixel receives as well input from a reflective HIP 161 and itssuccessive generations. The processed results of 165 and 161 areweighted, and then gathered in the image pixel 164.

Implementation. The core of the present invention is the DAS mechanism.When implemented in path tracing, it generates secondary rays andlocates their intersection with scene objects, excluding the use ofaccelerating structures of prior art. The DAS mechanism, which is basedon a conventional raster graphics pipeline, is implementable either by aGPU hardware pipeline, or by a CPU software pipeline. The parallelstructure of GPU makes it more efficient for graphics pipeline thangeneral-purpose CPU. A GPU is a specialized electronic circuit designedto accelerate the graphics pipeline. Where a CPU consists of a few coresfocused on sequential serial processing, GPUs pack thousands of smallercores designed for multitasking. There are two main types of graphicsprocessor: integrated and discrete. The DAS can be either utilized by aseparate component in a system (discrete GPU), or by an embedded GPU onthe CPU die (integrated GPU). Integrated GPU is used in embeddedsystems, mobile phones, personal computers, workstations, and gameconsoles.

The computing tasks that create the augmented objects and their visualcontext within the preset scene, as described in a great detailhereinbefore, are mostly based on graphics pipeline. For these tasks,the use of GPU is a great advantage. There is also additional task ofcollecting the sampled values of global illumination, processing thesevalues according to the rendering equation, and converging the resultsin the image pixels. The collecting task, being associated withconventional processing, can be implemented either by a CPU or by aGPGPU. There is also an additional task, associated with user's viewingdevice 171, shown in FIG. 17. For augmented reality those are wearablecomputer glasses that add information alongside or to what the wearersees. Typically this is achieved through an optical head-mounted display(OH MD) or embedded wireless glasses with transparent heads-up display(HUD) or AR overlay that has the capability of reflecting projecteddigital images as well as allowing the user to see through it. Forvirtual reality, the viewing device 171 can represent a virtual realityheadset that provides virtual reality for the wearer. VR headsets arewidely used with computer games but they are also used in otherapplications, including simulators and trainers. They comprise astereoscopic head-mounted display (providing separate images for eacheye), stereo sound, and head motion tracking sensors. Either way,component 171 must be interfaced to the computing platform by an APIsoftware, which is typically implemented by a CPU.

Consequently, the embodiments of the present invention call for acombined implementation of CPUs and GPUs, as generally shown in FIG. 17.The GPUs may stand for discrete graphics, integrated graphics or acombination of both: integrated graphics teaming with a discretegraphics.

Integrated graphics means that the GPU is integrated onto the CPU dieand shares memory with the processor. Since integrated GPUs rely on thesystem RAM, they don't have the computing power of their discretecounterparts that are contained on their own card and come equipped withtheir own memory, VRAM. Integrated GPU has a lower memory bandwidth fromsystem RAM, compared to discrete graphics cards between their VRAM andGPU core. This bandwidth is what is referred to as the memory bus andcan be performance limiting. Moreover, as a GPU is extremely memoryintensive, integrated processing may find itself competing with the CPUfor the relatively slow system RAM, as it has minimal or no dedicatedvideo memory. For the best possible graphics performance, discretegraphics chips outperform integrated GPUs.

On the other hand, sharing the same RAM memory can also be a benefit, asthe graphics cores on a multicore chip can better collaborate with CPUcores for exchange big data. The pure graphics tasks of imaging objects,reflections and color bleeding produce big data of light values, thatmust be collected and calculated for rendering equation by the CPUcores.

However, despite the performance advantage of discrete GPU, it isdesirable to use integrated GPU for implementation of the presentinvention in such applications as augmented reality, virtual reality,and computer games, for its better power-efficiency, affordability,portability and versatility. Integrated GPU, as a constituent of amulticore CPU chip, is used in embedded systems, mobile phones, tabletsand game consoles.

In addition to using discrete GPU or integrated GPU, it is also analternative of using a hybrid system having discrete and integratedGPUs, collaborating and alternating, depending on the task.

What is claimed is:
 1. A method for rendering an augmented object in athree-dimensional scene, the method comprising the following steps: a)shooting primary rendering projections at an augmented object by meansof a graphics pipeline; b) generating a cluster of primary hit points;c) determining a sub-scene space to be targeted by secondary rays; d)setting a reference point for secondary rendering projections; e)repeating multiple times, when each time the secondary renderingprojection is randomly tilted: 1) shooting secondary renderingprojection from vicinity of the reference point, through the cluster ofthe primary hit points, generating render target texture of thedetermined sub-scene; 2) seeking for intersection points of thesecondary rays with the sub-scene; 3) saving the intersection points fornext generation secondary rays; 4) sampling light values at theintersection points; and 5) feeding the sampled values back to theprimary hit points; f) processing the aggregated values in the primaryhit points; and g) converging the processing results into image pixels.2. The method of claim 1, wherein the sub-scene can be arbitrarilychosen as part of the three-dimensional scene.
 3. The method of claim 1,wherein more than one sub-scene can be determined.
 4. The method ofclaim 1, wherein each sub-scene must be targeted by separate sets ofsecondary rays.
 5. The method of claim 4, wherein for each sub-scene adifferent reference point is set.
 6. The method of claim 1, wherein saidsub-scene can be clipped from the three-dimensional scene.
 7. The methodof claim 1, wherein the shifting of secondary rendering projection isdone randomly.
 8. The method of claim 1, wherein the secondary renderingprojection can be of a perspective shape.
 9. The method of claim 1,wherein the secondary rendering projection can be of a parallel shape.10. The method of claim 1, wherein the result of the secondary renderingprojection is a render target texture of the determined sub-scene. 11.The method of claim 1, wherein the seeking for an intersection between asecondary ray and the determined sub-scene is done by matchingcoordinates of the primary hit point and the render target texture. 12.The method of claim 1, wherein the saved intersection points ofsecondary rays can serve as a cluster of primary hit points for the nextgeneration of secondary rays.
 13. The method of claim 1, wherein theprocessed samples of color and light values aggregated from all theprimary hit points, are converged into the image pixels, creating a fullimage of the augmented object affected by the three-dimensional scene.